[OpenVINO] Support Gemma3TextModel for feature-extraction by mlukasze · Pull Request #1799 · huggingface/optimum-intel

mlukasze · 2026-06-17T12:57:01Z

What does this PR do?

Adds a feature-extraction inputs override to Gemma3TextOpenVINOConfig so that Gemma3TextModel-based text-embedding models (e.g. microsoft/harrier-oss-v1-270m) export cleanly to OpenVINO IR.

Without the override the config inherits GemmaOpenVINOConfig.inputs, which injects position_ids into the graph and breaks sentence-transformers-style callers that pass only input_ids + attention_mask.

Mirrors the pattern used for Qwen3OpenVINOConfig added in #1415.

Installation instructions

pip install git+https://github.com/mlukasze/optimum-intel.git@enable/microsoft-harrier-oss-v1-270m
pip install -U openvino openvino-tokenizers nncf
pip install transformers

Exporting cmd-line

optimum-cli export openvino -m microsoft/harrier-oss-v1-270m ov_harrier_270m --task feature-extraction
optimum-cli export openvino -m microsoft/harrier-oss-v1-270m ov_harrier_270m_int8 --task feature-extraction --weight-format int8

Inference script

from transformers import AutoTokenizer
from optimum.intel import OVModelForFeatureExtraction

model_id = "microsoft/harrier-oss-v1-270m"
model_dir = "ov_harrier_270m_int8"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForFeatureExtraction.from_pretrained(model_dir)

sentences = ["OpenVINO accelerates inference.", "Text embeddings with Gemma3."]
inputs = tokenizer(sentences, return_tensors="pt", padding=True)

outputs = model(**inputs)
embeddings = outputs.last_hidden_state[:, 0]  # CLS pooling
print(embeddings.shape)  # torch.Size([2, 1152])

Before submitting

[N/A] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

rkazants

needs proper PR description with code snippets

HuggingFaceDocBuilderDev · 2026-06-17T17:01:42Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

rkazants · 2026-07-01T08:37:49Z

@mitruska, please check failing test https://github.com/huggingface/optimum-intel/actions/runs/28499616266/job/84473639987?pr=1799

mlukasze · 2026-07-07T16:26:46Z

Thanks for flagging this, @mitruska — investigated and pushed a fix in 2d0d8d4.

Root cause (two distinct issues, both external to this PR's diff)

build (*modeling*, 4.57.6) — AssertionError: False is not true
tiny-random-gemma3-text stores model.*-prefixed weights (a Gemma3ForCausalLM-style save). On transformers 4.57.6, Gemma3TextModel.base_model_prefix resolves to "" (not "model"), so none of the checkpoint's weights match on load — the reference model silently falls back to random init (transformers logs "newly initialized" for every param). Verified with a determinism test: two independent loads produce different embed_tokens.weight values. Comparing OV output against a randomly-initialized reference is meaningless.

build (*modeling*, latest) — RuntimeError: Float did not match BFloat16
On latest, base_model_prefix correctly resolves to "model" so weights load fine, but AutoModel.from_pretrained now honors the checkpoint's torch_dtype: bfloat16, so the reference runs in bf16 while OV always computes in fp32 (F32_CONFIG). torch.allclose raises on the dtype mismatch before any value comparison. Cast to fp32 manually: max abs diff is 0.0148 — normal bf16 rounding noise.

Neither issue is caused by this PR's diff (only touches Gemma3TextOpenVINOConfig.inputs) — both stem from the tiny checkpoint's structure vs. transformers-version-dependent Gemma3TextModel internals. Real checkpoints like microsoft/harrier-oss-v1-270m are saved unwrapped (architectures: ["Gemma3TextModel"], no prefix) and are unaffected — consistent with the ~0.999 WWB similarity already measured for this model.

Fix

Narrowly scoped to model_arch == "gemma3_text" in test_compare_to_transformers (zero effect on other archs):

Skip with a clear message when base_model_prefix != "model" (checkpoint weights don't/can't load) — same pattern already used elsewhere in this file (ticket 175062).
Otherwise, compare in fp32 with atol=2e-2 (vs default 1e-4) to absorb bf16 rounding, mirroring the existing per-arch atol overrides for flaubert/squeezebert.

Verification (local)

transformers 4.57.6: target test now skips with a clear reason (was: AssertionError)
transformers latest (5.13.0): target test now passes for real, not just skipped (was: RuntimeError)
Full OVModelForFeatureExtractionIntegrationTest class, both versions: no regressions for any other architecture
Confirmed test_compare_to_transformers_4_qwen3_vl_embedding (latest) and the test_quantization.py PIL.UnidentifiedImageError failures are pre-existing/unrelated (reproduced identically with git stash, i.e. present without this change too) — out of scope for this fix

Follow-up (logged, not addressed here to keep this PR minimal)

The durable fix is re-saving tiny-random-gemma3-text as a bare Gemma3TextModel (unprefixed, no baked-in torch_dtype) to match real checkpoints structurally. I don't have push access to optimum-intel-internal-testing, so I've prepared (and verified) a ready-to-run script for a maintainer: happy to share if useful — just let me know.

CI on the new commit is currently action_required (needs a maintainer to approve running workflows on this fork PR) — flagging in case that needs a manual click on your end.

Enables clean OpenVINO feature-extraction export of Gemma3 text-embedding backbones such as microsoft/harrier-oss-v1-270m. Mirrors Qwen3OpenVINOConfig feature-extraction support (huggingface#1415): excludes position_ids and exports only input_ids + attention_mask for the embedding path. - model_configs.py: inputs property override on Gemma3TextOpenVINOConfig - test_modeling.py: gemma3_text in OVModelForFeatureExtractionIntegrationTest - test_export.py: gemma3_text in ExportModelTest Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…eights tiny-random-gemma3-text on the Hub stores Gemma3ForCausalLM weights (with model. prefix) while model_type=gemma3_text. Loading via AutoModel/ Gemma3TextModel causes ALL weights to be randomly re-initialised, so the OV-exported model and the PyTorch reference end up with *different* random seeds and their outputs diverge → test_compare_to_transformers fails. Add _create_tiny_gemma3_text_model() that creates a Gemma3TextModel from the Hub config, saves its weights to a temp dir, and returns the path. MODEL_NAMES[gemma3_text] now points at this locally-saved model, so both the export and the comparison load identical weights. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>

…egression) Address review feedback from rkazants on PR huggingface#1799: - Replace the atol=2e-2 workaround with forcing the reference PyTorch model to load in fp32 (torch_dtype=torch.float32), matching the precedent already used for AutoModelForCausalLM elsewhere in this test suite. gemma3_text now uses the same tight atol=1e-4 as every other architecture instead of a special-cased tolerance. - Replace the vague external-issue-reference skip reason with a concrete, verified explanation: Gemma3TextModel.base_model_prefix resolves to "language_model" on transformers==4.50.0, "" on 4.53.0/4.55.0, and only becomes "model" starting with transformers==5.0.0, which is why AutoModel.from_pretrained silently falls back to random weights on older releases instead of raising. This was empirically verified across several transformers releases.

rkazants marked this pull request as ready for review June 17, 2026 16:59

rkazants reviewed Jun 17, 2026

View reviewed changes

rkazants requested review from echarlaix and popovaan June 17, 2026 17:00

mlukasze changed the title ~~[EXPERIMENT][WIP] Enable OpenVINO feature-extraction export for gemma3_text (Gemma3TextModel) embedding backbones~~ [OpenVINO] Support Gemma3TextModel for feature-extraction Jun 18, 2026

mlukasze requested a review from rkazants June 18, 2026 04:51

rkazants approved these changes Jun 19, 2026

View reviewed changes

rkazants reviewed Jul 1, 2026

View reviewed changes

Comment thread tests/openvino/utils_tests.py Outdated

rkazants reviewed Jul 1, 2026

View reviewed changes

Comment thread tests/openvino/utils_tests.py Outdated

rkazants reviewed Jul 1, 2026

View reviewed changes

Comment thread tests/openvino/utils_tests.py Outdated

rkazants reviewed Jul 8, 2026

View reviewed changes

Comment thread tests/openvino/test_modeling.py Outdated

rkazants reviewed Jul 8, 2026

View reviewed changes

Comment thread tests/openvino/test_modeling.py Outdated

omega-intel and others added 5 commits July 14, 2026 13:28

Apply suggestions from code review

c3d0fad

Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>

Apply suggestion from @rkazants

e6f8acb

omega-intel force-pushed the enable/microsoft-harrier-oss-v1-270m branch from 2d0d8d4 to 65bddbb Compare July 14, 2026 11:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[OpenVINO] Support Gemma3TextModel for feature-extraction#1799

[OpenVINO] Support Gemma3TextModel for feature-extraction#1799
mlukasze wants to merge 5 commits into
huggingface:mainfrom
mlukasze:enable/microsoft-harrier-oss-v1-270m

mlukasze commented Jun 17, 2026 •

edited

Loading

Uh oh!

rkazants left a comment

Uh oh!

HuggingFaceDocBuilderDev commented Jun 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rkazants commented Jul 1, 2026

Uh oh!

mlukasze commented Jul 7, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mlukasze commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Installation instructions

Exporting cmd-line

Inference script

Before submitting

Uh oh!

rkazants left a comment

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Jun 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rkazants commented Jul 1, 2026

Uh oh!

mlukasze commented Jul 7, 2026

Root cause (two distinct issues, both external to this PR's diff)

Fix

Verification (local)

Follow-up (logged, not addressed here to keep this PR minimal)

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mlukasze commented Jun 17, 2026 •

edited

Loading